Quantized Adam with Error Feedback

In this article, we present a distributed variant of an adaptive stochastic gradient method for training deep neural networks in the parameter-server model. To reduce communication cost among workers and server, incorporate two types quantization schemes, i.e., weight quantization, into proposed Adam. addition, to bias introduced by operations, propose error-feedback technique compensate quantized gradient. Theoretically, nonconvex setting, show that with error feedback converges first-order stationary point, point related level under both single-worker multi-worker modes. Last, apply methods train networks. Experimental results demonstrate efficacy our methods.